Skip to content

fix electric collection progressive mode #852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Nov 21, 2025
Merged

fix electric collection progressive mode #852

merged 7 commits into from
Nov 21, 2025

Conversation

@samwillis
Copy link
Collaborator

@samwillis samwillis commented Nov 19, 2025 •
edited
Loading

Depends on electric-sql/electric#3463, must have deps bumped to new electric client version when that is released. done

Fix progressive mode to use fetchSnapshot and atomic swap

Progressive mode was broken because requestSnapshot() injected snapshots into the stream in causally correct position, which didn't work properly with the full mode stream. This release fixes progressive mode by:

Core Changes:

  • Use fetchSnapshot() during initial sync to fetch and apply snapshots immediately in sync transactions
  • Buffer all stream messages during initial sync (renamed flag to isBufferingInitialSync)
  • Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready
  • Track txids/snapshots only after atomic swap (enables correct optimistic transaction confirmation)

Test Infrastructure:

  • Added ELECTRIC_TEST_HOOKS symbol for test control (hidden from public API)
  • Added progressiveTestControl.releaseInitialSync() to E2E test config for explicit transition control
  • Created comprehensive progressive mode E2E test suite (8 tests):
    • Explicit snapshot phase and atomic swap validation
    • Txid tracking behavior (Electric-only)
    • Multiple concurrent snapshots with deduplication
    • Incremental updates after swap
    • Predicate handling and resilience tests

Bug Fixes:

  • Fixed type errors in test files
  • All 166 unit tests + 95 E2E tests passing

@changeset-bot
Copy link

changeset-bot bot commented Nov 19, 2025 •
edited
Loading

🦋 Changeset detected

Latest commit: 2846a6e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages
Name Type
@tanstack/electric-db-collection Patch
@tanstack/db-collection-e2e Patch
@tanstack/db Patch
@tanstack/angular-db Patch
@tanstack/powersync-db-collection Patch
@tanstack/react-db Patch
@tanstack/rxdb-db-collection Patch
@tanstack/solid-db Patch
@tanstack/svelte-db Patch
@tanstack/trailbase-db-collection Patch
@tanstack/vue-db Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@samwillis samwillis marked this pull request as draft November 19, 2025 12:18
@samwillis samwillis force-pushed the samwillis/fix-electric-progressive branch from 85a5bd7 to b702326 Compare November 19, 2025 12:33
@pkg-pr-new
Copy link

pkg-pr-new bot commented Nov 19, 2025 •
edited
Loading

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@852

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@852

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@852

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@852

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@852

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@852

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@852

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@852

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@852

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@852

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@852

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@852

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@852

commit: 2846a6e

@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025 •
edited
Loading

Size Change: 0 B

Total Size: 86.2 kB

ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.38 kB
./packages/db/dist/esm/collection/changes.js 977 B
./packages/db/dist/esm/collection/events.js 388 B
./packages/db/dist/esm/collection/index.js 3.24 kB
./packages/db/dist/esm/collection/indexes.js 1.1 kB
./packages/db/dist/esm/collection/lifecycle.js 1.67 kB
./packages/db/dist/esm/collection/mutations.js 2.26 kB
./packages/db/dist/esm/collection/state.js 3.43 kB
./packages/db/dist/esm/collection/subscription.js 2.42 kB
./packages/db/dist/esm/collection/sync.js 2.29 kB
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.19 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/index.js 2.64 kB
./packages/db/dist/esm/indexes/auto-index.js 742 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 1.87 kB
./packages/db/dist/esm/indexes/lazy-index.js 1.1 kB
./packages/db/dist/esm/indexes/reverse-index.js 513 B
./packages/db/dist/esm/local-only.js 837 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.22 kB
./packages/db/dist/esm/query/builder/functions.js 733 B
./packages/db/dist/esm/query/builder/index.js 3.96 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 917 B
./packages/db/dist/esm/query/compiler/evaluators.js 1.35 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 1.8 kB
./packages/db/dist/esm/query/compiler/index.js 1.96 kB
./packages/db/dist/esm/query/compiler/joins.js 2 kB
./packages/db/dist/esm/query/compiler/order-by.js 1.25 kB
./packages/db/dist/esm/query/compiler/select.js 1.07 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 673 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 5.26 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 1.74 kB
./packages/db/dist/esm/query/live/internal.js 130 B
./packages/db/dist/esm/query/optimizer.js 2.56 kB
./packages/db/dist/esm/query/predicate-utils.js 2.88 kB
./packages/db/dist/esm/query/subset-dedupe.js 921 B
./packages/db/dist/esm/scheduler.js 1.21 kB
./packages/db/dist/esm/SortedMap.js 1.18 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 881 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 852 B
./packages/db/dist/esm/utils/index-optimization.js 1.51 kB
./packages/db/dist/esm/utils/type-guards.js 157 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025 •
edited
Loading

Size Change: 0 B

Total Size: 3.34 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 225 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.17 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.11 kB
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 431 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

@samwillis samwillis force-pushed the samwillis/fix-electric-progressive branch from 3941a16 to 132b57f Compare November 19, 2025 14:17
@samwillis samwillis marked this pull request as ready for review November 19, 2025 14:17
KyleAMathews
KyleAMathews approved these changes Nov 19, 2025
View reviewed changes
Copy link
Collaborator

@KyleAMathews KyleAMathews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! All that work you did on truncate is coming in handy! Very nice semantics to be able to easily do this clean swap.

@kevin-dp
Copy link
Contributor

kevin-dp commented Nov 20, 2025

Mostly note to self during review:

Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready

--> Need to check that we also cancel in-flight fetchSnapshot requests and/or ignore fetchSnapshot requests that resolve after the swap.

kevin-dp
kevin-dp previously requested changes Nov 20, 2025
View reviewed changes
Copy link
Contributor

@kevin-dp kevin-dp left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great work, i think we're almost there but i believe there is a bug with fetchSnapshot requests that resolve after the up-to-date message. For the rest i only have minor comments/nits.

@samwillis samwillis force-pushed the samwillis/fix-electric-progressive branch from a24fe7f to 2846a6e Compare November 21, 2025 11:32
@samwillis
Copy link
Collaborator Author

samwillis commented Nov 21, 2025

@kevin-dp I've addressed your review. We should still come back to this file and tidy it up more.

@samwillis samwillis requested a review from kevin-dp November 21, 2025 11:36
@samwillis samwillis force-pushed the samwillis/fix-electric-progressive branch from 8fc3efd to 2846a6e Compare November 21, 2025 19:14
@samwillis samwillis dismissed kevin-dp’s stale review November 21, 2025 19:35

Discussed with Kyle, merging as is before the weekend. We can make further changes later.

@samwillis samwillis merged commit acb3e4f into main Nov 21, 2025
13 checks passed
@samwillis samwillis deleted the samwillis/fix-electric-progressive branch November 21, 2025 19:35
@github-actions github-actions bot mentioned this pull request Nov 21, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KyleAMathews KyleAMathews KyleAMathews approved these changes

@kevin-dp kevin-dp Awaiting requested review from kevin-dp

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@samwillis @kevin-dp @KyleAMathews